Image processing device, image processing method, and computer program

ABSTRACT

An image processing device includes an edge strength calculation unit, and a mixing unit. The edge strength calculation unit calculates an edge strength of a pixel of interest in an input image from a stairs tone strength of the pixel of interest and a gradient of the pixel of interest, wherein the gradient is calculated from pixel values of the pixel of interest and pixels adjacent to the pixel of interest, and the stairs tone strength is indicative of differences between the gradient and a variance of the pixel of interest calculated from the pixel values of the pixel of interest and the adjacent pixels. The mixing unit calculates an illumination light component of the pixel of interest of the input image from an average value of the pixel of interest and pixel values adjacent to the pixel of interest, and a mixing ratio calculated from the edge strength.

CROSS-REFERENCE TO RELATED APPLICATIONS

This U.S. non-provisional patent application claims priority under 35U.S.C. §119 from, and the benefit of, Japanese Patent Application No.2015-033766, filed on Feb. 24, 2015, the contents of which are hereinincorporated by reference in their entirety.

BACKGROUND

Embodiments of the present disclosure are directed to an imageprocessing device, an image processing method, and an image processingprogram.

Imaging devices, such as digital still cameras, are widely used. Adynamic range compression technology such as high dynamic range (HDR) orbacklight compensation is used in some imaging devices. Such imagingdevices compress a range of one or both of a highlight area and a shadowarea of an image of a subject for which a brightness range is wider thana dynamic range of an imaging element, to acquire an image.

Furthermore, a technology for improving visibility of an image having acompressed brightness range uses image processing based on retinextheory on the image to perform local dynamic range compensation. Basedon retinex theory, brightness components of an input image are dividedinto illumination light components and reflectivity components, and theillumination light components are modulated and then combined with thereflectivity components, to obtain an output image having a locallycompensated dynamic range.

Based on retinex theory, a smoothing filter can be applied to the inputimage to extract the illumination light components. Due to theapplication of the smoothing filter to the input image, an unnaturalimage pattern not originally present in the image, i.e., a halophenomenon, may form in a boundary region between a subject and abackground.

An ε filter can be applied as a smoothing filter and a value of ε iscontrolled, so that smoothing is performed while an edge is maintained.Based on an application of this smoothing filter, a boundary between abackground and a subject, referred to as a “stairs tone edge”, is notdifferentiated from a boundary in which contrast (gradation) is changedby a subject's shape, referred to as a “pattern tone edge”. Therefore,for example, even if a subject is uniformly with an illumination thatdoes not depend on the subject's shape, an edge may be maintained at ahigh contrast portion and the shape itself may be treated as anillumination light component. Therefore, an image free from a halophenomenon may be obtained, but contrast may be deteriorated withrespect to the subject's shape.

SUMMARY

Embodiments of the present disclosure can provide an image processingdevice, an image processing method, and an image processing program fordifferentiating a stairs tone edge from a pattern tone edge in an imageto perform a nonlinear smoothing operation on the image.

An embodiment of the inventive concept provides an image processingdevice including an edge strength calculation unit that calculates anedge strength of a pixel of interest in an input image from a stairstone strength of the pixel of interest and a gradient of the pixel ofinterest, wherein the gradient is calculated from pixel values of thepixel of interest and pixels adjacent to the pixel of interest, and thestairs tone strength is indicative of differences between the gradientand a variance of the pixel of interest calculated from the pixel valuesof the pixel of interest and the adjacent pixels, and a mixing unit thatcalculates an illumination light component of the pixel of interest ofthe input image from the pixel value of the pixel of interest, anaverage value of the pixel values of the pixels adjacent to the pixel ofinterest, and a mixing ratio of the pixel of interest calculated fromthe edge strength.

In an embodiment, the image processing device may include a variancecalculation unit that calculates the average value from the pixel valuesof the pixel of interest and adjacent pixels within a predeterminedrange of the pixel of interest, and calculates the variance from theaverage value.

In an embodiment, the image processing device may include a gradientcalculation unit that calculates the gradient of the pixel of interestfrom pixel values of the of the pixel of interest and adjacent pixelswithin a predetermined range of the pixel of interest.

In an embodiment, the image processing device may include a mixing ratiocalculation unit that calculates the mixing ratio of the pixel ofinterest from the edge strength of the pixel of interest.

In an embodiment, the mixing ratio K_(E) is calculated from

${f_{E}\left( K_{G} \right)} = \left\{ {\begin{matrix}1 & \ldots & {K_{G\;} < {th}_{1}} \\\min & \ldots & {K_{G} \geq {th}_{2}} \\{\frac{\left( {1 - \min} \right)\left( {{{th}\;}_{2} - K_{G}} \right)}{{th}_{\; 2} - {th}_{1}} + \min} & \ldots & {otherwise}\end{matrix},} \right.$

wherein K_(G) is the edge strength, f_(E)(K_(G)) is a function thatconverts the edge strength K_(G) into the mixing ratio K_(E), th₁ andth₂ are threshold values of the edge strength, wherein th₁<th₂, and minis a preset constant greater than zero.

In an embodiment, the stairs tone strength K_(S)(x, y) of the pixel ofinterest is calculated from wherein

${K_{S}\left( {x,y} \right)} = \left\{ {\begin{matrix}{{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}} & \ldots & {{{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}} \leq 1} \\1 & \ldots & {otherwise}\end{matrix},} \right.$

wherein ∇²(x, y) is the gradient of the pixel of interest, and ρ²(x, y)is the variance of the pixel of interest, and the edge strength K_(G)(x,y) of the pixel of interest is calculated from K_(G)(x, y)=α·∇²(x,y)·K_(S), wherein α is a predetermined constant.

In an embodiment, the mixing unit may include a first multiplicationunit that multiplies the pixel value of the pixel of interest by a firstcoefficient based on the mixing ratio, a second multiplication unit thatmultiplies the average value by a second coefficient based on the mixingratio, a subtraction unit that that calculates that first coefficient bysubtracting the mixing ratio from one, and an addition unit that adds anoutput of the first multiplication unit to an output of the secondmultiplication unit.

In an embodiment, the image processing device may include a divisionunit that receives the illumination light component from theillumination light generation unit and that calculates a reflectivitycomponent of the input image by dividing the input image by theillumination light component, an illumination light modulation unit thatlocally modulates the illumination light component to generate a newillumination light component, and a multiplication unit that multipliesthe reflectivity component received from the division unit by the newillumination light component received from the illumination lightmodulation unit to generate a brightness component.

In an embodiment of the inventive concept, an image processing methodincludes calculating an edge strength of a pixel of interest in an inputimage from a stairs tone strength of the pixel of interest and agradient of the pixel of interest, wherein the gradient is calculatedfrom pixel values of the pixel of interest and pixels adjacent to thepixel of interest, and the stairs tone strength is indicative ofdifferences between the gradient and a variance of the pixel of interestcalculated from the pixel values of the pixel of interest and theadjacent pixels, and calculating an illumination light component of thepixel of interest of the input image from the pixel value of the pixelof interest, an average value of the pixel values of the pixels adjacentto the pixel of interest, and a mixing ratio calculated from the edgestrength.

In an embodiment, the average value may be calculated from the pixelvalues of the pixel of interest and adjacent pixels within apredetermined range of the pixel of interest, and the variance iscalculated from the average value.

In an embodiment, the gradient may be calculated from pixel values ofthe pixel of interest and adjacent pixels within a predetermined rangeof the pixel of interest.

In an embodiment, the mixing ratio K_(E) of the pixel of interest may becalculated from the edge strength of the pixel of interest using

${f_{E}\left( K_{G} \right)} = \left\{ {\begin{matrix}1 & \ldots & {K_{G\;} < {th}_{1}} \\\min & \ldots & {K_{G} \geq {th}_{2}} \\{\frac{\left( {1 - \min} \right)\left( {{{th}\;}_{2} - K_{G}} \right)}{{th}_{\; 2} - {th}_{1}} + \min} & \ldots & {otherwise}\end{matrix},} \right.$

wherein K_(G) is the edge strength, f_(E)(K_(G)) is a function thatconverts the edge strength K_(G) into the mixing ratio K_(E), th₁ andth₂ are threshold values of the edge strength, wherein th₁<th₂, and minis a preset constant greater than zero.

In an embodiment, the stairs tone strength K_(S)(x, y) of the pixel ofinterest may be calculated from

${K_{S}\left( {x,y} \right)} = \left\{ {\begin{matrix}{{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}} & \ldots & {{{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}} \leq 1} \\1 & \ldots & {otherwise}\end{matrix},} \right.$

wherein ∇²(x, y) is the gradient of the pixel of interest, and σ²(x, y)is the variance of the pixel of interest, and the edge strength K_(G)(x,y) of the pixel of interest is calculated from K_(G)(x, y)=α·∇²(x,y)·K_(S), wherein α is a predetermined constant.

In an embodiment, in the illumination light component L(x, y) of thepixel of interest may be calculated from L(x, y)=(1−K_(E))·I(x,y)+K_(E)·A(x, y), wherein K_(E) is the mixing ratio, I(x, y) is thepixel value of the input image at the pixel of interest, and A(x, y) isthe average value of the pixel of interest.

In an embodiment of the inventive concept, a non-transitory programstorage device readable by a computer, tangibly embodying a program ofinstructions executed by the computer to perform the method steps forprocessing an image, the method comprising the steps of calculating anedge strength of a pixel of interest in an input image from a stairstone strength of the pixel of interest and a gradient of the pixel ofinterest, wherein the gradient is calculated from pixel values of thepixel of interest and pixels adjacent to the pixel of interest, and thestairs tone strength is indicative of differences between the gradientand a variance of the pixel of interest calculated from the pixel valuesof the pixel of interest and the adjacent pixels, and calculating anillumination light component of the pixel of interest of the input imagefrom the pixel value of the pixel of interest, an average value of thepixel value of pixels adjacent to the pixel of interest, and a mixingratio of the pixel of interest calculated from the edge strength.

In an embodiment, the average value may be calculated from the pixelvalues of the pixel of interest and adjacent pixels within apredetermined range of the pixel of interest, and the variance iscalculated from the average value.

In an embodiment, the gradient may be calculated from pixel values ofthe pixel of interest and adjacent pixels within a predetermined rangeof the pixel of interest.

In an embodiment, the mixing ratio K_(E) of the pixel of interest may becalculated from the edge strength of the pixel of interest using

, wherein K_(G) is the edge strength, f_(E)(K_(G)) is a function thatconverts the edge strength K_(G) into the mixing ratio K_(E), th₁ andth₂ are threshold values of the edge strength, wherein th₁<th₂, and minis a preset constant greater than zero.

In an embodiment, the stairs tone strength K_(S)(x, y) of the pixel ofinterest may be calculated from

, wherein ∇²(x, y) is the gradient of the pixel of interest, and σ²(x,y) is the variance of the pixel of interest, and the edge strengthK_(G)(x, y) of the pixel of interest is calculated from K_(G)(x,y)=α·∇²(x, y)·K_(S), wherein α is a predetermined constant.

In an embodiment, in the illumination light component L(x, y) of thepixel of interest may be calculated fromL(x,y)=(1−K_(E))·I(x,y)+K_(E)·A(x,y), wherein K_(E) is the mixing ratio,I(x, y) is the pixel value of the input image at the pixel of interest,and A(x, y) is the average value of the pixel of interest.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an exemplary configuration of an imageprocessing device according to an embodiment of the inventive concept.

FIG. 2 illustrates an exemplary result of a smoothing operationperformed by an illumination light generation unit.

FIG. 3 is a block diagram of an exemplary configuration of anillumination light generation unit according to a comparative example.

FIG. 4 illustrates an exemplary function that performs edge strength—εvalue conversion by an ε value adjusting unit.

FIG. 5 illustrates an example of an input II.

FIG. 6 is a graph that illustrates a change in a gradient for each pixelof an input image when a gradient is calculated by applying a primarydifferential filter to the input illustrated in FIG. 5.

FIG. 7 is a block diagram of an exemplary configuration of anillumination light generation unit according to an embodiment of theinventive concept.

FIG. 8 is a graph that illustrates an example of a change in a standarddeviation based on a variation for each pixel of an input imagecalculated on the basis of the input illustrated in FIG. 5.

FIG. 9 is a graph that illustrates an example of a change in a stairstone strength for each pixel of an input image calculated on the basisof the input illustrated in FIG. 5.

FIG. 10 is a graph that illustrates an example of a change in an edgestrength for each pixel of an input image calculated on the basis of theinput illustrated in FIG. 5.

FIG. 11 illustrates a process of adjusting an ε value by an ε valueadjusting unit according to an embodiment.

FIG. 12 illustrates an exemplary function for setting the ε value by theε value adjusting unit according to the edge strength.

FIG. 13 illustrates another exemplary function for setting the ε valueby the ε value adjusting unit according to the edge strength.

FIG. 14 illustrates another exemplary function for setting the ε valueby the ε value adjusting unit according to the edge strength.

FIG. 15 is a flowchart of an operation method of an illumination lightgeneration unit according to an embodiment.

FIG. 16 is a block diagram of an exemplary configuration of anillumination light generation unit according to an embodiment of theinventive concept.

FIG. 17 illustrates an example of a conversion function that performsedge strength-mixing ratio conversion.

FIG. 18 illustrates another example of a conversion function thatperforms edge strength-mixing ratio conversion.

FIG. 19 illustrates a schematic configuration of a mixing unit.

FIG. 20 is a flowchart of an operation method of an illumination lightgeneration unit according to an embodiment; and

FIG. 21 is a diagram that illustrates an estimated circuit size of anexemplary illumination light generation unit according to an embodimentof the inventive concept.

DETAILED DESCRIPTION

Exemplary embodiments of the inventive concept will be described indetail with reference to the accompanying drawings. In the descriptionand the drawings, elements that have substantially the sameconfiguration may be referred to by the same reference numeral to avoidoverlapping descriptions.

1. Overview: Image Processing Device Based on Retinex Theory

Image processing based on retinex theory is briefly described below toassist with an understanding of an image processing device according toan embodiment of the inventive concept.

In general, when an image of a subject is acquired by an imaging devicesuch as a digital camera, a brightness range of natural light on thesubject may exceed a dynamic range of an imaging element in the imagingdevice. Therefore, some imaging devices employ a dynamic rangecompression technology such as high dynamic range (HDR) or backlightcompensation to acquire an image of a subject of which a brightnessrange is wider than a dynamic range of an imaging element. Imagingdevices that use dynamic range compression technology compress the rangeof one or both of a highlight area and a shadow area of an acquiredimage, thereby enabling acquisition of an image of a subject for which abrightness range is wider than a dynamic range of an imaging element.

According to a technology for improving image visibility by compressinga dynamic range, local dynamic range compensation based on retinextheory can be performed as an image processing operation.

In detail, according to retinex theory, light in an image can beregarded as a product of an illumination light component and areflectivity component. Letting the illumination light component be Land the reflectivity component be R, a brightness I of an input imagecan be expressed as Equation (1), below:

I=L×R.   (1)

Hereinafter, for convenience, I, L and R are respectively referred to asII, LL and RR.

When local dynamic range compensation based on retinex theory isperformed, an image processing device separately processes theillumination light component LL and the reflectivity component RR fromthe brightness component II of the input image. Hereinafter, thebrightness II of the input image may be simply referred to as an “inputII”.

An exemplary configuration of an image processing device for performinglocal dynamic range compensation based on retinex theory according to anembodiment of the inventive concept will be described with reference toFIG. 1. FIG. 1 is a block diagram of an exemplary configuration of animage processing device according to an embodiment of the inventiveconcept.

As illustrated in FIG. 1, an image processing device 1 according to apresent embodiment includes an illumination light generation unit 10, adivision unit 30, an illumination light modulation unit 50, and amultiplication unit 70.

The illumination light generation unit 10 generates the illuminationlight component LL from the input II. In detail, the illumination lightgeneration unit 10 performs a smoothing operation on the input II, forexample, by applying a smoothing filter, to generate the illuminationlight component LL from the input II. Furthermore, the illuminationlight generation unit 10 outputs, to the division unit 30 and theillumination light modulation unit 50, the generated illumination lightcomponent LL. The illumination light generation unit 10 will bedescribed in more detail below.

The division unit 30 generates the reflectivity component RR from theinput II. In detail, the division unit 30 receives the illuminationlight component LL from the illumination light generation unit 10, andcalculates the reflectivity component RR by dividing the input II by theillumination light component LL on the basis of Equation (1). Thedivision unit 30 outputs the reflectivity component RR to themultiplication unit 70.

The illumination light modulation unit 50 receives, from theillumination light generation unit 10, the illumination light componentLL generated from the input II. The illumination light modulation unit50 locally modulates the illumination light component LL to generate anew illumination light component LL′. Furthermore, the illuminationlight modulation unit 50 outputs the new illumination light componentLL′ to the multiplication unit 70.

The multiplication unit 70 multiplies the reflectivity component RRreceived from the division unit 30 by the new illumination lightcomponent LL′ received from the illumination light modulation unit 50,i.e., the new illumination light component LL′ obtained by locallymodulating the illumination light component LL. The multiplication unit70 outputs as an output image, to a predetermined external device, animage based on a brightness component II′ generated by recombining thereflectivity component RR and the illumination light component LL′.

As described above, the image processing device 1 generates and outputsan output image obtained by compensating a dynamic range of an inputimage.

2. An Embodiment

2.1. Overview

An illumination light generation unit 10 according to an embodiment ofthe inventive concept will be described.

As described above, the illumination light generation unit 10 performs asmoothing operation on the input II by applying a smoothing filter togenerate the illumination light component LL from the input II. Due tothe smoothing operation performed on the input II, a halo phenomenon mayoccur. A halo phenomenon occurs when a large brightness change at asubject-background boundary is smoothed by the smoothing operation andcontrast of a portion adjacent to the boundary is reduced.

For example, FIG. 2 illustrates an exemplary result of a smoothingoperation performed by the illumination light generation unit 10. Indetail, FIG. 2 illustrates a change in brightness intensity for eachpixel of an image with respect to each of an input and an output whenthe smoothing operation is performed on a subject-background boundary.Herein, the brightness intensity may represent, for example, a pixelvalue, a luminance value, or a lightness value. In the descriptionbelow, the term “brightness intensity” may correspond to any one of theforgoing values. In the graph of FIG. 2, the horizontal axis representsa pixel location in an image and the vertical axis represents thebrightness intensity. Furthermore, in the graph of FIG. 2, referencesign II represents a brightness component of an input image, andreference sign LL represents an output, i.e., a generated illuminationlight component, when the smoothing operation is performed on the inputII.

As illustrated in FIG. 2, once the smoothing operation is performed atthe subject-background boundary in the input II, the illumination lightcomponent LL is generated in which a rapid brightness change at theboundary has been reduced. As the rapid brightness change between thesubject and the background is reduced, the contrast of thesubject-background boundary is reduced, which may induce a halophenomenon.

As a comparative example, an example of the illumination lightgeneration unit 10 which generates the illumination light component LLwithout generating a halo phenomenon will be described with reference toFIG. 3. FIG. 3 is a block diagram of an exemplary configuration of anillumination light generation unit according to a comparative example.An illumination light generation unit according to a comparative exampleis referred to reference number 10′ to differentiate it from theillumination light generation unit 10 according to a present embodiment.

In the illumination light generation unit 10′ according to a comparativeexample illustrated in FIG. 3, an ε filter is used for performing asmoothing operation, and an ε value of the ε filter is controlled toperform the smoothing operation while maintaining an edge with respectto the input II.

In detail, as illustrated in FIG. 3, the illumination light generationunit 10′ according to a comparative example includes a smoothingprocessing unit 11 and an ε value control unit 19. The smoothingprocessing unit 11 corresponds to the ε filter. The value control unit19 includes a gradient calculation unit 191 and an ε value adjustingunit 193.

The gradient calculation unit 191 calculates a gradient ∇ for each pixelof interest on the basis of the brightness intensity of pixels adjacentto the pixel of interest, in which each pixel of the input image II issequentially processed as the pixel of interest.

As an example of a method for calculating the gradient ∇ by the gradientcalculation unit 191, a primary differential filter, i.e., a high-passfilter, can be applied as expressed by Equation (2), below:

|∇|(x,y)=|I(x−n,y)−|I(x+n,y)|+|I(x,y−n)−I(x,y+n)|  (2)

In Equation (2), n denotes an operator length for specifying an adjacentpixel. Furthermore, I(x−n, y) and I(x+n, y) denote a brightnesscomponent of an adjacent pixel positioned in an x direction, such as ahorizontal direction, with respect to a pixel of interest, for anoperator length of n. Likewise, I(x, y−n) and I(x, y+n) denote abrightness component of an adjacent pixel positioned in a y direction,such as a vertical direction, with respect to a pixel of interest, foran operator length of n. In addition, a tap number is 3 when theoperator length n=1, and the tap number is 5 when the operator lengthn=2.

As another example, the gradient calculation unit 191 may calculate thegradient using a band limiting filter as expressed by Equation (3)below:

$\begin{matrix}{{{\nabla }\left( {x,y} \right)} = {{{\sum\limits_{j = {- n}}^{+ n}\left\lbrack {{I\left( {{x - n},{y + j}} \right)} - {I\left( {{x + n},{y + j}} \right)}} \right\rbrack}} + {{\sum\limits_{i = {- n}}^{+ n}\left\lbrack {{I\left( {{x + i},{y - n}} \right)} - {I\left( {{x + i},{y + n}} \right)}} \right\rbrack}}}} & (3)\end{matrix}$

As described above, for each pixel of interest, the gradient calculationunit 191 calculates the gradient ∇ on the basis of a brightnessintensity of pixels adjacent to the pixel of interest. Furthermore, thegradient calculation unit 191 outputs, to the ε value adjusting unit193, the gradient ∇ calculated for each pixel of interest.

The ε value adjusting unit 193 receives, from the gradient calculationunit 191, the gradient ∇ calculated for each pixel of interest of aninput image. The ε value adjusting unit 193 considers the gradient ∇, ormore specifically, an absolute value of the gradient ∇, received foreach pixel of interest to be an edge strength K_(G) of each pixel ofinterest. The edge strength K_(G)(x,y) of the pixel of interest can bederived from Equation (4) below, letting the coordinates of the pixel ofinterest be (x,y) and the gradient of the pixel of interest be ∇(x,y):

K _(c)(x, y)=|∇(x, y)|.   (4)

Furthermore, the ε value adjusting unit 193 converts the edge strengthK_(G) into the ε value to perform edge strength—ε value conversion, sothat the ε value decreases as the edge strength K_(G) increases, or theε value increases as the edge strength K_(G) decreases. For example,FIG. 4 illustrates an example of a function fε(K_(G)) for performingedge strength—ε value conversion by the ε value adjusting unit 193.

Accordingly, the ε value adjusting unit 193 calculates, for each pixelof interest, the ε value from the edge strength K_(G) of the pixel ofinterest, and outputs the calculated ε value to the smoothing processingunit 11.

The smoothing processing unit 11 receives, from the ε value adjustingunit 193, the ε value calculated for each pixel of interest of an inputimage, and applies an ε filter to the pixel of interest and pixelsadjacent thereto based on the acquired ε value. As described above, thesmoothing processing unit 11 performs a smoothing operation by applyingthe ε filter to the input II, and outputs as the illumination lightcomponent LL a brightness component obtained by performing the smoothingoperation.

By virtue of the above-described configuration, the illumination lightgeneration unit 10′ according to a comparative example performs anonlinear smoothing operation on the input II to reduce the smoothingeffect for an edge portion to maintain the edge, but increases thesmoothing effect for portions other than an edge. In this manner, theillumination light generation unit 10′ according to a comparativeexample can generate the illumination light component LL withoutgenerating a halo phenomenon.

According to the illumination light generation unit 10′ according to acomparative example, a subject-background boundary, also referred to asa “stairs tone edge” below, is not substantially differentiated from aboundary in which the contrast (gradation) is changed by the subject'sshape, also referred to as a “pattern tone edge” below. Therefore, theillumination light generation unit 10′ equally calculates an edgestrength K_(G)G without differentiating a stairs tone edge from apattern tone edge, and performs a smoothing operation according to theedge strength K_(G).

For example, FIG. 5 illustrates an example of an input II, morespecifically, an example of changes in brightness intensities of pixelsof an input image. For ease of description, FIG. 5 illustrates changesin brightness intensities of pixels of an input image with respect to anx direction, such as a horizontal direction, of the input image. Thatis, the horizontal axis of the graph of FIG. 5 indicates a pixellocation in the input image with respect to the x direction.Furthermore, the vertical axis of the graph of FIG. 5 indicates abrightness intensity of each pixel.

FIG. 6 illustrates an example of a change in the gradient ∇ for eachpixel of the input image when the gradient ∇ is calculated by applying aprimary differential filter to the input II illustrated in FIG. 5. Thatis, the horizontal axis of the graph of FIG. 6 corresponds to thehorizontal axis of the graph of FIG. 5 and indicates a pixel location inthe input image with respect to the x direction. Furthermore, thevertical axis of the graph of FIG. 6 indicates the gradient ∇ calculatedfor each pixel.

In FIG. 5, portions v11 a, v11 b, v13 a, v13 b, v15 a, and v15 brepresent portions of the input image in which contrast changes due to astairs tone edge. The portion v17 represents a portion in which contrastchanges due to a pattern tone edge. Furthermore, in FIG. 6, the portionsv11 a, v11 b, v13 a, v13 b, v15 a, v15 b, and v17 correspond to the samelocations of the input image indicated by the same reference numbers inFIG. 5.

In FIG. 6, stairs tone edges v15 a and v15 b and a pattern tone edge v17b have substantially the same gradient ∇ value. Therefore, in the casewhere the gradient ∇ is an edge K_(G), the illumination light generationunit 10′ does not substantially differentiate the stairs tone edges v15a and v15 b from the pattern tone edge v17 b.

Here, a portion where contrast changes due to a subject's shape, i.e., aportion of a pattern tone, can be regarded as being uniformlyilluminated regardless of the shape, and, in some cases, the patterntone portion can undergo a smoothing operation without maintaining anedge. However, the illumination light generation unit 10′ according to acomparative example maintains not only a stairs tone edge but also apattern tone edge. Therefore, when an input image is processed by theillumination light generation unit 10′ according to a comparativeexample, a blurred output image may be output due to the subject'scontrast being compressed by a smoothing operation.

The illumination light generation unit 10 according to a presentembodiment can differentiate a stairs tone edge from a pattern tone edgein an input image, thereby enabling nonlinear smoothing processing forthe input image. The illumination light generation unit 10 according toa present embodiment is described in more detail below.

2.2. Function Configuration

An exemplary configuration of the illumination light generation unit 10according to an embodiment of the inventive concept will be describedwith reference to FIG. 7. FIG. 7 is a block diagram of an exemplaryconfiguration of the illumination light generation unit 10 according toa present embodiment.

As illustrated in FIG. 7, the illumination light generation unit 10according to a present embodiment includes a smoothing processing unit11 and an ε value control unit 13. Since the smoothing processing unit11 is the same as that of the illumination light generation unit 10′described with respect to FIG. 3, the smoothing processing unit 11 isnot described in detail below. The ε value control unit 13 includes agradient calculation unit 131, a variance calculation unit 133, and an εvalue adjusting unit 135.

The gradient calculation unit 131 is the same as the gradientcalculation unit 191 of the illumination light generation unit 10′according to a comparative example as described above. That is, thegradient calculation unit 131 calculates a gradient ∇ for each pixel ofinterest from the brightness intensities of the pixels adjacent to thepixel of interest, wherein each pixel of the input image II issequentially processed as the pixel of interest.

Furthermore, the gradient calculation unit 131 calculates the gradient ∇for each pixel of interest by calculating a convolution integral using afilter operator. Equation (5), below, is an example of expressing theprimary differential filter of Equation (2) or (3) by a convolutionintegral.

f′ ^((x)) =∇f(x)=W

X,   (5)

In Equation (5), W denotes an operator for calculating the gradient ∇,also referred to as a “gradient operator”, below. Equation (6), below,is an example of the gradient operator W when an operator length n=1:

$\begin{matrix}{W = {\begin{bmatrix}{- 1} & 0 & 1\end{bmatrix} \cdot {\frac{1}{2}.}}} & (6)\end{matrix}$

For another example, Equation (7) below is an example of the gradientoperator W when the operator length n=2:

$\begin{matrix}{W = {\begin{bmatrix}{- 1} & {- 1} & 0 & 1 & 1\end{bmatrix} \cdot {\frac{1}{4}.}}} & (7)\end{matrix}$

The gradient operators of Equations (6) and (7) are merely examples, andthe operator length n and the operator coefficients may change in otherembodiments.

As described above, for each pixel of interest, the gradient calculationunit 131 calculates the gradient ∇ from the brightness intensities ofpixels adjacent to the pixel of interest. Furthermore, the gradientcalculation unit 131 outputs, to the ε value adjusting unit 135, thegradient ∇ calculated for each pixel of interest.

The variance calculation unit 133 calculates a variance σ² for eachpixel of interest from the brightness intensities of the pixel ofinterest and of pixels adjacent to the pixel of interest, where eachpixel of the input image II is sequentially processed as the pixel ofinterest.

Here, the variance σ²(x, y) of the pixel of interest is calculated usingEquation (8) below, letting the coordinates of the pixel of interest be(x, y) and the brightness intensity of a pixel located at thecoordinates (x−i, y−j) be I_(x−j, y−i):

$\begin{matrix}{{\sigma^{2}\left( {x,y} \right)} = {\frac{1}{\left( {{2\; m} + 1} \right)^{2}}{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- m}}^{m}\left( {I_{{x - i},{y - j}} - \overset{\_}{I}} \right)^{2}}}}} & (8) \\{\overset{\_}{I} = {\frac{1}{\left( {{2\; m} + 1} \right)^{2}}{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- m}}^{m}{I_{{x - i},{y - j}}.}}}}} & \;\end{matrix}$

Equation (8) may be expanded as expressed by Equation (9), below:

$\begin{matrix}\begin{matrix}{{\sigma^{2}\left( {x,y} \right)} = {\frac{1}{\left( {{2m} + 1} \right)^{2}}{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- m}}^{m}\left( {I_{{x - i},{y - j}} - \overset{\_}{I}} \right)^{2}}}}} \\{= {\frac{1}{\left( {{2m} + 1} \right)^{2}}{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- m}}^{m}\left( {I_{{x - i},{y - j}}^{2} - {2{I_{{x - i},{y - j}} \cdot \overset{\_}{I}}} + {\overset{\_}{I}}^{2}} \right)}}}} \\{= {{\frac{1}{\left( {{2m} + 1} \right)^{2}}{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- m}}^{m}I_{{x - i},{y - j}}^{2}}}} - \frac{1}{\left( {{2m} + 1} \right)^{2}}}} \\{{{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- m}}^{m}{2{I_{{x - i},{y - j}} \cdot \overset{\_}{I}}}}} + {\frac{1}{\left( {{2m} + 1} \right)^{2}}{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- m}}^{m}{\overset{\_}{I}}^{2}}}}}} \\{= {{\frac{1}{\left( {{2m} + 1} \right)^{2}}{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- m}}^{m}I_{{x - i},{y - j}}^{2}}}} - \frac{2\overset{\_}{I}}{\left( {{2m} + 1} \right)^{2}}}} \\{{{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- m}}^{m}I_{{x - i},{y - j}}}} + {\overset{\_}{I}}^{2}}} \\{= {{\frac{1}{\left( {{2m} + 1} \right)^{2}}{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- m}}^{m}I_{{x - i},{y - j}}^{2}}}} - {2\overset{\_}{II}} + {\overset{\_}{I}}^{2}}} \\{= {{\frac{1}{\left( {{2m} + 1} \right)^{2}}{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- m}}^{m}I_{{x - i},{y - j}}^{2}}}} - {\overset{\_}{I}}^{2}}}\end{matrix} & (9)\end{matrix}$

FIG. 8 is a graph that illustrates a change in a standard deviation abased on the variation σ² calculated for each pixel of the input imageII of FIG. 5. Furthermore, to match units with respect to the gradient ∇for each pixel illustrated in FIG. 6, the horizontal axis of the graphof FIG. 8 corresponds to the horizontal axes of the graphs of FIGS. 5and 6 and indicates a pixel location in the input image with respect tothe x direction. Furthermore, the vertical axis of the graph of FIG. 8indicates the standard deviation a calculated for each pixel. Thestandard deviation σ is calculated as the square root of the varianceσ². The variance calculation unit 133 may calculate the standarddeviation σ instead of the variance σ².

As described above, for each pixel of interest, the variance calculationunit 133 calculates the variance σ² from the brightness intensities ofthe pixel of interest and pixels adjacent to the pixel of interest.Furthermore, the variance calculation unit 133 outputs, to the ε valueadjusting unit 135, the variance σ² for each pixel of interest.

The ε value adjusting unit 135 receives, from the gradient calculationunit 131, the gradient ∇ for each pixel of interest. Furthermore, the εvalue adjusting unit 135 receives, from the variance calculation unit133, the variance σ² for each pixel of interest.

The ε value adjusting unit 135 according to a present embodiment will bedescribed by comparing the change of the gradient ∇ for each pixelillustrated in FIG. 6 with the change of the standard deviation σ, or,in other words, the variance σ², for each pixel illustrated in FIG. 8.

As described above with reference to FIG. 6, when considering thegradient ∇ for each pixel, the stairs tone edges v15 a and v15 b and thepattern tone edge v17 b have approximately the same gradient ∇ value,and are thus not substantially differentiated from each other.

Alternatively, when considering the standard deviation a for each pixel,as illustrated in FIG. 8, the stairs tone edges v15 a and v15 b and thepattern tone edge v17 b have different standard deviation a values, andthus can be differentiated from each other. In addition, the stairs toneedges v13 a and v13 b and the pattern tone edge v17 have approximatelythe same standard deviation σ value, and are thus not substantiallydifferentiated from each other.

By comparing FIGS. 6 and 8, it may be seen that the gradient ∇ differsfrom the standard deviation σ at portions where a stairs tone edge isnot substantially differentiated from a pattern tone edge. Thisdifference results from characteristic differences between the gradient∇ and the standard deviation σ. The ε value adjusting unit 135 accordingto a present embodiment can differentiate a stairs tone edge from apattern tone edge using the characteristic differences.

In detail, the gradient ∇ indicates a difference between pixels adjacentto or near a pixel of interest. In Equation (8), the standard deviationσ indicates a change in a range defined by an operator length m.Ideally, regarding a stairs tone edge, a maximum value (peak) of thestandard deviation σ tends to have approximately the same value as amaximum value of an absolute gradient ∇ value. However, regarding apattern tone edge, the standard deviation σ tends to have a larger valuethan that of the absolute gradient ∇ value. Therefore, the ε valueadjusting unit 135 according to a present embodiment calculates adiscrepancy between the gradient ∇ and the standard deviation σ as astairs tone strength K_(S), and differentiates a stairs tone edge from apattern tone edge using the stairs tone strength K_(S).

For example, letting the coordinates of a pixel of interest be (x, y),the stairs tone strength K_(S)(x, y) of the pixel of interest iscalculated from the gradient ∇(x, y) and the standard deviation G(x, y)of the pixel of interest using Equation (10) below:

$\begin{matrix}{{K_{S}\left( {x,y} \right)} = {\frac{k_{\nabla} \cdot {{\nabla\left( {x,y} \right)}}}{k_{\sigma} \cdot {\sigma \left( {x,y} \right)}} = {k \cdot {\frac{{\nabla\left( {x,y} \right)}}{\sigma \left( {x,y} \right)}.}}}} & (10)\end{matrix}$

In Equation (10), k_(∇) and k_(σ) are compensation (or normalization)coefficients for the gradient ∇(x, y) and the standard deviation σ(x, y)respectively and are determined so that a ratio between the absolutevalue of the ∇(x, y) and the absolute value of the standard deviationσ(x, y) is 1. Furthermore, the compensation coefficient k represents theratio of the coefficients k_(∇) and k_(σ).

The coefficients k_(∇) and k_(σ) may be pre-calculated integers. Foranother example, based on a plurality of ideal stairs tone edges withdifferent brightness intensities, a function for compensating |∇|/(σ)may be pre-calculated, and the function may be determined with thecompensation coefficients k_(∇) and k_(σ).

Equation (10) can be generalized to express the stairs tone strengthK_(S) as Equation (11), below:

$\begin{matrix}{K_{S} = {\frac{k_{\nabla} \cdot {\nabla }}{k_{\sigma} \cdot \sigma} = {k \cdot {\frac{\nabla }{\sigma}.}}}} & (11)\end{matrix}$

Here, the stairs tone strength K_(S) has a value of 1 or less, and, asthe stairs tone strength K_(S) approaches 1, brightness changes morerapidly in the vicinity of a corresponding pixel of interest. That is,as the stairs tone strength K_(S) approaches 1, the corresponding pixelof interest is more likely to correspond to a stairs tone edge.

The above-disclosed equation for calculating the stairs tone strengthK_(S) is merely an example. A method of calculating the stairs tonestrength K_(S) is not limited to the above example, as long as themethod indicates a degree of discrepancy between the gradient ∇ and thestandard deviation σ (or the variance σ²).

As a specific example, the ε value adjusting unit 135 can calculate thestairs tone strength K_(S) from the discrepancy between the square ofthe gradient ∇ and the variance σ². In this case, letting thecoordinates of a pixel of interest be (x, y), the stairs tone strengthK_(S)(x, y) of the pixel of interest can be calculated from the gradient∇(x, y) and the standard deviation σ(x, y) of the pixel of interest asexpressed by Equation (12) below:

$\begin{matrix}{{K_{s}\left( {x,y} \right)} = {\frac{k_{\nabla} \cdot {\nabla^{2}\left( {x,y} \right)}}{k_{\sigma} \cdot {\sigma^{2}\left( {x,y} \right)}} = {k \cdot {\frac{\nabla^{2}\left( {x,y} \right)}{\sigma^{2}\left( {x,y} \right)}.}}}} & (12)\end{matrix}$

Equation (12) can be generalized to express the stairs tone strengthK_(S) as Equation (13), below:

$\begin{matrix}{K_{s} = {\frac{k_{\nabla} \cdot \nabla^{2}}{k_{\sigma} \cdot \sigma^{2}} = {k \cdot {\frac{\nabla^{2}}{\sigma^{2}}.}}}} & (13)\end{matrix}$

As another example, the ε value adjusting unit 135 can calculate thestairs tone strength K_(S) from the discrepancy using a differencebetween the square of the gradient ∇ and the variance σ². In this case,letting the coordinates of a pixel of interest be (x, y), the stairstone strength K_(S)(x, y) for the pixel of interest can be calculatedfrom the gradient ∇(x, y) and the variance σ²(x, y) of the pixel ofinterest as expressed by Equation (14), below:

K _(S)(x,y)=k _(σ)·σ²(x,y)−k _(∇)·∇²(x,y).   (14)

Equation (14) can be generalized to express the stairs tone strengthK_(S) as Equation (15), below:

K _(S) =k _(σ)·σ² −k _(∇)·∇².   (15)

As described above, the ε value adjusting unit 135 calculates the stairstone strength K_(S) based on the discrepancy between the gradient ∇ andthe standard deviation σ (or the variance σ²). For example, FIG. 9illustrates a change in the stairs tone strength K_(S) for each pixel ofthe input image II illustrated in FIG. 5, i.e., calculated on the basisof the gradient ∇ for each pixel illustrated in FIG. 6 and the standarddeviation σ for each pixel illustrated in FIG. 8. That is, thehorizontal axis of the graph of FIG. 9 corresponds to the horizontalaxes of the graphs of FIGS. 5, 6 and 8 and indicates a pixel location inthe input image with respect to the x direction. Furthermore, thevertical axis of the graph of FIG. 9 indicates the stairs tone strengthK_(S) calculated for each pixel.

As shown in FIG. 9, stairs tone edges v11 a, v11 b, v13 a, v13 b, v15 a,and v15 b differ from pattern zone edge v17 with respect to the stairstone strength K_(S).

As a specific example, when the stairs tone strength K_(S) iscalculated, an operator length n for calculating the gradient ∇ isinitialized to the same value as an operator length m for calculatingthe variance σ². In this case, regarding a stairs tone edge, even if thecompensation coefficients k_(∇) and k_(σ) are set to be 1, there islittle difference between the absolute value of the gradient ∇ and thestandard deviation σ, and the stairs tone strength K_(S) isapproximately 1. On the other hand, regarding a pattern tone edge, theabsolute value of the gradient ∇ is less than the standard deviation σ,and the stairs tone strength K_(S) has a lesser value than that for astairs tone edge.

On the basis of such characteristics, the ε value adjusting unit 135calculates, for each pixel of interest of an input image, an edgestrength K_(G) for a pattern tone edge by multiplying the absolute valueof the gradient ∇ of the pixel of interest by the stairs tone strengthK_(S) calculated for the pixel of interest. That is, letting thecoordinates of a pixel of interest be (x, y), the edge strength K_(G)(x, y) of the pixel of interest is calculated from the gradient ∇(x, y)and the stairs tone strength K_(S) of the pixel of interest as expressedby Equation (16) below:

K _(G)(x, y)=|∇(x, y)|·K _(S).   (16)

The operator length n for calculating the gradient ∇ and the operatorlength m for calculating the variance σ² may be set such that n≦m.

As another example, the ε value adjusting unit 135 can calculate, foreach pixel of interest of an input image, the edge strength K_(G) bymultiplying a gradient ∇² of the pixel by the stairs tone strength K_(S)as expressed by Equation (17) below:

K _(G)(x, y)=∇²(x, y)·K _(S)   (17)

As described above, the ε value adjusting unit 135 calculates, for eachpixel of interest of an input image, the edge strength K_(G) from thegradient ∇ of the pixel of interest and the stairs tone strength K_(S)of the pixel of interest. For example, FIG. 10 illustrates a change inthe edge strength K_(G) for each pixel of the input image II illustratedin FIG. 5, i.e., calculated from the gradient ∇ for each pixelillustrated in FIG. 6 and the stairs tone strength K_(S) for each pixelillustrated in FIG. 9. That is, the horizontal axis of the graph of FIG.10 corresponds to the horizontal axes of the graphs of FIGS. 5, 6 and 9and indicates a pixel location in the input image with respect to the xdirection. Furthermore, the vertical axis of the graph of FIG. 10indicates the edge strength K_(G) calculated for each pixel.

As shown in FIG. 10, since the edge strength K_(G) is calculated fromthe stairs tone strength K_(S) and the gradient ∇ for each pixel ofinterest, a pattern tone edge v17 is suppressed, and stairs patternedges v11 a, v11 b, v13 a, v13 b, v15 a, and v15 b can be extractedusing the edge strength K_(G). That is, the ε value adjusting unit 135adjusts the ε value on the basis of the edge strength K_(G) calculatedas described above performs edge strength—ε value conversion, so that,between a stairs tone edge and a pattern tone edge, the stairs tone edgeis maintained.

The adjustment of the ε value performed by the ε value adjusting unit135 based on the edge strength K_(G) will be described in detail withreference to FIG. 11. FIG. 11 illustrates a process of adjusting the εvalue by the ε value adjusting unit 135 according to a presentembodiment.

The upper graph of FIG. 11 illustrates a change in brightness strengthfor each pixel of an input image. The lower graph of FIG. 11 illustratesa change in the edge strength K_(G) for each pixel of interest of theinput image. The horizontal axes of each of the upper and lower graphsof FIG. 11 indicate a pixel location in the input image with respect tothe x direction. Furthermore, the vertical axis of the upper graph ofFIG. 11 indicates a brightness intensity, and the vertical axis of thelower graph of FIG. 11 indicates the edge strength K_(G).

In FIG. 11, n represents an operator length, and it is assumed that theoperator length n for calculating the gradient ∇ and the operator lengthm for calculating the variance σ² satisfy the relation of m=n.Furthermore, in FIG. 11, pixel location (x)=0 corresponds to a stairstone edge, and the locations of x=±1, ±2 correspond to adjacent pixelsalong the x direction, assuming that a pixel at location (x)−0 is apixel of interest.

In the example illustrated in FIG. 11, a halo phenomenon may occur wherethe pixel location (x) ranges from −2 to +2, assuming that the operatorlength n of an ε filter is 2. Therefore, in the example illustrated inFIG. 11, the ε value of the ε filter can decreased, i.e., minimized,where the pixel location (x) ranges from −2 to +2.

Therefore, as illustrated in the lower graph of FIG. 11, the ε valueadjusting unit 135 sets a threshold value th_(G) of the edge strengthK_(G) and compares the edge strength K_(G) calculated for each pixel ofinterest with the threshold value th_(G). For example, the ε valueadjusting unit 135 can set the ε value to a minimum value ε_(MIN) for apixel whose edge strength K_(G) exceeds the threshold value th_(G), andsets the ε value to a maximum value ε_(MAX) for a pixel whose edgestrength K_(G) is less than or equal to the threshold value th_(G).

For example, FIG. 12 illustrates an exemplary function f_(ε)(K_(G)) forperforming edge strength—ε value conversion by setting, by the ε valueadjusting unit 135, the ε value based on the edge strength K_(G).

For comparison, FIG. 13 illustrates another exemplary functionf_(ε)(K_(G)) for performing the edge strength—ε value conversion by theε value adjusting unit 135. A comparative example of FIG. 13 illustratesa function f_(ε)(K_(G)) that performs edge strength—ε value conversionusing the same method as the ε value adjusting unit 193 of theillumination light generation unit 10′ of FIG. 2.

As illustrated in FIG. 13, the ε value adjusting unit 193 according to acomparative example performs linear modulation in a portion between themaximum and minimum values ε_(MAX) and ε_(MIN) of the ε value that arefunctions of the threshold values th_(G1) and th_(G2). This is becausethe ε value should be uniformly controlled by the edge strength K_(G)without differentiating a stairs tone edge from a pattern tone edge,since it is challenging for the ε value adjusting unit 193 todifferentiate a stairs tone edge from a pattern tone edge.

Due to such characteristics, according to the ε value adjusting unit 193according to a comparative example, a high edge strength pattern toneedge may be maintained without being smoothed, and a low edge strengthstairs tone edge may be smoothed and not maintained. Furthermore,according to the ε value adjusting unit 193 according to a comparativeexample, for example, a smoothing operation is performed on a pixel witha relatively low edge strength than an intermediate edge strength, asindicated at positions x=±1, ±2 in FIG. 11. Therefore, a pixel's edgemaintenance is decreased, and a weak halo phenomenon may occur in thevicinity of the pixel due to the smoothing operation.

Regarding this issue, the ε value adjusting unit 135 according to anembodiment adjusts the ε value based on the edge strength K_(G)calculated on the basis of the stairs tone strength K_(S) and thegradient ∇ for each pixel of interest. As described above, the edgestrength K_(G) has a high value for a stairs tone edge and a low valuefor a pattern tone edge. Therefore, the ε value adjusting unit 135 canperform edge strength—ε value conversion by threshold processing basedon the threshold value th_(G) as illustrated in FIG. 12.

Furthermore, due to the above-mentioned characteristics of the edgestrength K_(G), the value adjusting unit 135 according to a presentembodiment may set the threshold value th_(G) to a relatively low value,lower than at least the threshold value th_(G2) shown in FIG. 13.Therefore, according to the ε value adjusting unit 135 according to apresent embodiment, an edge can be maintained even for a pixel with arelatively low edge strength indicated at positions x=±1, ±2 in FIG. 11.

Furthermore, the function f_(ε)(K_(G)) of FIG. 12 is merely an example,and the ε value adjusting unit 135 according to a present embodiment mayapply the function f_(ε)(K_(G)) of FIG. 13 to the edge strength K_(G)calculated from the stairs tone strength K_(S) and the gradient ∇ foreach pixel of interest. For another example, the ε value adjusting unit135 may apply the function f_(ε)(K_(G)) of FIG. 14 to the edge strengthK_(G) calculated from the stairs tone strength K_(S) and the gradient ∇for each pixel of interest, to perform edge strength—ε value conversion.FIG. 14 illustrates another example of the function f_(ε)(K_(G)) forperforming edge strength—ε value conversion by setting, by the ε valueadjusting unit 135, the ε value based on the edge strength K_(G).

As described above, the ε value adjusting unit 135 sets the ε value foreach pixel of interest based on the edge strength K_(G) calculated foreach pixel of interest of an input image, and outputs the ε value to thesmoothing processing unit 11.

The following process is the same as that described above with respectto the illumination light generation unit 10′ according to a comparativeexample. That is, the smoothing processing unit 11 receives, from the εvalue adjusting unit 135, the value calculated for each pixel ofinterest of an input image, and applies an ε filter to the pixel ofinterest and pixels adjacent thereto based on the acquired ε value. Asdescribed above, the smoothing processing unit 11 performs a smoothingoperation by applying the ε filter to the input II, and outputs abrightness component obtained after performing the smoothing operationas the illumination light component LL.

2.3. Processing

An operation method of the illumination light generation unit 10according to a present embodiment will be described with reference toFIG. 15. FIG. 15 is a flowchart of an operation method of theillumination light generation unit 10 according to an embodiment.

Operation S101

The gradient calculation unit 131 calculates the gradient ∇ for eachpixel of interest from a brightness intensity of pixels adjacent to thepixel of interest, where each pixel of the input image II issequentially processed as the pixel of interest. Furthermore, thegradient calculation unit 131 may calculate the gradient ∇ for eachpixel of interest by calculating a convolution integral using a filteroperator. The gradient ∇ may be calculated on the basis of any one ofEquations (2) and (3). Furthermore, the gradient calculation unit 131outputs, to the ε value adjusting unit 135, the gradient ∇ for eachpixel of interest.

Operation S103

The variance calculation unit 133 calculates a variance σ² for eachpixel of interest from the brightness intensities of the pixel ofinterest and of pixels adjacent to the pixel of interest, where eachpixel of the input image II is sequentially processed as the pixel ofinterest. The variance σ² may be calculated from Equation (8).Furthermore, the variance calculation unit 133 outputs, to the ε valueadjusting unit 135, the variance σ² for each pixel of interest.

Operation S105

The ε value adjusting unit 135 receives, from the gradient calculationunit 131, the gradient ∇ for each pixel of interest, and, from thevariance calculation unit 133, the variance σ² for each pixel ofinterest. The ε value adjusting unit 135 calculates the stairs tonestrength K_(S) based on the discrepancy between the gradient ∇ and thestandard deviation σ for each pixel of interest. The stairs tonestrength K_(S) may be calculated using any one of Equations (10), (12)and (14).

Operation S107

Thereafter, the ε value adjusting unit 135 calculates, for each pixel ofinterest, the edge strength K_(G) from the gradient ∇ for the pixel ofinterest and the stairs tone strength K_(S) for the pixel of interest.The edge strength K_(G) may be calculated on the basis of any one ofEquations (16) and (17).

Operation S109

Once the edge strength K_(G) is calculated for each pixel of interest,the ε value adjusting unit 135 compares the edge strength K_(G) with thepredetermined threshold value th_(G) for each pixel of interest, andsets the ε value for the pixel of interest according to the comparisonresult. As a specific example, based on the function f_(ε)(K_(G)) ofFIG. 12, the ε value adjusting unit 135 sets the ε value to the minimumvalue ε_(MIN) for a pixel for which the edge strength K_(G) exceeds thethreshold value th_(G), and sets the ε value to the maximum valueε_(MAX) for a pixel for which the edge strength K_(G) is less than orequal to the threshold value th_(G)thG.

As described above, the ε value adjusting unit 135 sets the ε value foreach pixel of interest abased on the edge strength K_(G) calculated foreach pixel of interest of an input image, and outputs the set ε value tothe smoothing processing unit 11.

The smoothing processing unit 11 receives, from the ε value adjustingunit 135, the ε value for each pixel of interest of an input image, andapplies an ε filter to the pixel of interest and pixels adjacent theretoon the basis of the acquired ε value. As described above, the smoothingprocessing unit 11 performs a smoothing operation by applying the εfilter to the input II, and outputs a brightness component that resultsfrom the smoothing operation as the illumination light component LL.

An example of an “image processing method” corresponds to theabove-described series of operations in which the edge strength K_(G) iscalculated from the stairs tone strength KS, which measures thediscrepancy between the gradient ∇ and the variance σ², and a smoothingoperation is performed on the input image based on an ε value set basedon the edge strength K_(G).

The above-mentioned series of operations can be performed by a programthat processes each element of the display device 10 and executes on acentral processing unit (CPU). This program may be configured to beexecuted by an operating system (OS) installed in the device. A storagelocation of the program is not limited as long as it is readable by adevice that includes a CPU for performing the above-describedprocessing. For example, the program may be stored in an externalrecording medium accessible from the device. In this case, the recordingmedium in which the program is stored may be accessed by the device sothat the CPU of the device can execute the program.

2.4. Summary

As described above, the illumination light generation unit 10 accordingto an embodiment of the inventive concept calculates the gradient ∇ andthe variance σ² (or the standard deviation σ) for each pixel ofinterest, and calculates the stairs tone strength K_(S) based on thediscrepancy between the gradient ∇ and the variance σ². Furthermore, theillumination light generation unit 10 calculates the edge strength K_(G)based on the stairs tone strength K_(S) and the gradient ∇, and adjuststhe ε value based on the edge strength K_(G). As described above, theedge strength K_(G) has a high value for a stairs tone edge and has alow value for a pattern tone edge. Therefore, the illumination lightgeneration unit 10 can differentiate a stairs tone edge from a patterntone edge in an input image, and can perform a nonlinear smoothing thatreduces smoothing for a stairs tone edge and increases smoothing forother portions, including a pattern tone edge.

3. Another Embodiment

3.1. Overview

An illumination light generation unit according to an embodiment of theinventive concept will be described. In the illumination lightgeneration unit 10 according to an embodiment of FIG. 7, an output valueof an ε filter is used as an illumination light component.

When a reference pixel range of the filter is relatively wide, aweighted average coefficient based on a Gaussian distribution may beused as a smoothing coefficient. As the reference pixel range of the εfilter increases, the processing cost increases, which can increase acircuit size.

Here, to assist in understanding an illumination light generation unitaccording to a present embodiment, a process performed by the ε filterto calculate an illumination light component LL(x, y) on the basis of apixel value I_(x, y) of a pixel of interest (x, y) is expressed asEquation (18) below:

$\begin{matrix}{{{L\left( {x,y} \right)} = {I_{x,y} + {\sum\limits_{i = {- n}}^{n}\; {\sum\limits_{j = {- n}}^{n}\; {W_{i,j}{F\left( {I_{{x - i},{y - j}} - I_{x,y}} \right)}}}}}}{{\sum\limits_{i = {- n}}^{n}\; {\sum\limits_{j = {- n}}^{n}\; W_{i,j}}} = 1}{{F\left( {\pm z} \right)} = \left\{ {\begin{matrix}{\pm z} & {{\ldots \mspace{14mu} {z}} \leq ɛ} \\0 & {\ldots \mspace{14mu} {otherwise}}\end{matrix}.} \right.}} & (18)\end{matrix}$

In Equation (18), the coefficient W is a weighted average filtercoefficient that applies a weight so that the sum total of coefficientsis 1. The function F(z) is a smoothing filter so that a differencebetween values of a pixel of interest and a reference pixel is added tothe pixel of interest. According to Equation (18), if a difference valueis larger than the ε value, reference pixel information is not output asa result, so that an edge is maintained.

As described above, since the ε filter requires the calculation of thedifference value between a pixel of interest and each reference pixeland a comparison of the difference value with the ε value, a relativelylarge cost is incurred, and a circuit size may be increased.

Portable terminals such as smartphones are currently widely used, andsuch terminals are limited in terms of a size of a circuit includedtherein.

Described below is an illumination light generation unit according toanother embodiment of the inventive concept which can differentiate astairs tone edge from a pattern zone edge to perform nonlinearprocessing and further enables a reduction in circuit size. Hereinafter,an illumination light generation unit according to a present embodimentmay be referred to by reference number 20 to be differentiated from theillumination light generation unit 10 according to an embodiment of FIG.7.

3.2. Function Configuration

An exemplary configuration of the illumination light generation unit 20according to a present embodiment will be described with reference toFIG. 16. FIG. 16 is a block diagram of an exemplary configuration of theillumination light generation unit 20 according to a present embodiment.

As illustrated in FIG. 16, the illumination light generation unit 20according to a present embodiment includes a mixing unit 21 and a mixingratio control unit 23. The mixing ratio control unit 23 includes agradient calculation unit 231, a variance calculation unit 233, an edgestrength calculation unit 235, and a mixing ratio calculation unit 237.

The gradient calculation unit 231 calculates the square of a gradient ∇for each pixel of interest from brightness intensities of pixelsadjacent to the pixel of interest using a gradient operator W, whereeach pixel of the input image II is sequentially processed as the pixelof interest. The gradient calculation unit 231 calculates the gradient ∇in the same manner as the gradient calculation unit 131 according to anembodiment of FIG. 7. Equation (19) shown below is an example of agradient operator W when the operator length n=2:

$\begin{matrix}{W = {\begin{bmatrix}{- 1} & {- 1} & 0 & 1 & 1\end{bmatrix} \cdot {\frac{1}{4}.}}} & (19)\end{matrix}$

Equation (20) shown below is another example of the gradient operator Wwhen the operator length n=2:

$\begin{matrix}{W = {\begin{bmatrix}{- 1} & {- 1} & 0 & 1 & 1 \\{- 1} & {- 1} & 0 & 1 & 1 \\{- 1} & {- 1} & 0 & 1 & 1 \\{- 1} & {- 1} & 0 & 1 & 1 \\{- 1} & {- 1} & 0 & 1 & 1\end{bmatrix} \cdot {\frac{1}{20}.}}} & (20)\end{matrix}$

The gradient operators W in Equations (19) and (20) are merely examples,and embodiments are not limited thereto. That is, as a pass band isadjusted, coefficients other than those of Equation (19) or (20) may beused for the gradient operator W.

Furthermore, the gradient calculation unit 231 can calculate ∇², thesquare of the gradient ∇, for each pixel of interest by calculating aconvolution integral using the gradient operator W. Hereinafter, thesquare ∇² of the gradient ∇ will be referred to as a “gradient ∇²”.Equation (21), below, is an example of a formula for calculating thegradient ∇² by calculating a convolution integral using the gradientoperator W. In Equation (21), W′ denotes a coefficient obtained byrotating the gradient operator W by 90 degrees.

∇²(x,y)={(W

I)²+(W

I)²}/2   (21)

As described above, for each pixel of interest, the gradient calculationunit 231 calculates the gradient ∇² from the brightness intensities ofpixels adjacent to the pixel of interest. Furthermore, the gradientcalculation unit 231 outputs, to the edge strength calculation unit 235,the gradient ∇² for each pixel of interest.

The variance calculation unit 233 calculates, for each pixel ofinterest, an average value AA of the brightness intensities of the pixelof interest and of pixels adjacent to the pixel of interest, i.e.,calculates a moving average, and calculates the variance σ² for eachpixel from the average value AA, where each pixel of the input image IIis sequentially processes as the pixel of interest.

Here, letting the coordinates of the pixel of interest be (x, y), theoperator length be n, and the brightness intensity of a pixel positionedat the coordinates (x, y) be I_(x, y), the average value A(x, y) isexpressed by convolution integral shown in Equation (22) below:

$\begin{matrix}{{A\left( {x,y} \right)} = {{\frac{1}{\left( {{2\; n} + 1} \right)^{2}}{\sum\limits_{i = {- n}}^{n}\; {\sum\limits_{j = {- n}}^{n}\; I_{{x - i},{y - j}}}}} = {\begin{bmatrix}1 & 1 & 1 & 1 & 1 \\1 & 1 & 1 & 1 & 1 \\1 & 1 & 1 & 1 & 1 \\1 & 1 & 1 & 1 & 1 \\1 & 1 & 1 & 1 & 1\end{bmatrix} \cdot {\frac{1}{25} \otimes {I.}}}}} & (22)\end{matrix}$

The operator length n, i.e., the reference pixel range, for calculatingthe average value AA may be set based on a circuit size of a device inwhich the illumination light generation unit 20 is disposed.

The variance σ² for the pixel of interest (x, y) is calculated usingEquation (23) below, where the coordinates of the pixel of interest are(x, y) and the brightness intensity of a pixel located at thecoordinates (x−i, y−j) is I_(x−j, y−i):

$\begin{matrix}{{\sigma^{2}\left( {x,y} \right)} = {{\frac{1}{\left( {{2\; n} + 1} \right)^{2}}{\sum\limits_{i = {- n}}^{n}\; {\sum\limits_{j = {- n}}^{n}\; I_{{x - i},{y - j}}^{2}}}} - {{A^{2}\left( {x,y} \right)}.}}} & (23)\end{matrix}$

As described above, for each pixel of interest, the variance calculationunit 233 calculates the average value AA from the brightness intensitiesof the pixel of interest and of pixels adjacent to the pixel ofinterest, and calculates the variance σ² from the average value AA foreach pixel of interest. Furthermore, the variance calculation unit 233outputs, to the edge strength calculation unit 235, the variance σ² foreach pixel of interest. Furthermore, while calculating the variance σ²,the variance calculation unit 233 outputs, to the mixing unit 21, theaverage value AA, i.e., a moving average, calculated for each pixel ofinterest.

The edge strength calculation unit 235 receives, from the gradientcalculation unit 231, the gradient ∇² for each pixel of interest, and,from the variance calculation unit 233, the variance σ² for each pixelof interest. The edge strength calculation unit 235 calculates thediscrepancy between the gradient ∇² and the variance σ² as the stairstone strength K_(S).

For example, letting the coordinates of a pixel of interest be (x, y),the stairs tone strength K_(S) (x, y) of the pixel of interest iscalculated from the gradient ∇²(x, y) and the variance σ²(x, y) of thepixel of interest using Equation (24), below:

$\begin{matrix}{{K_{s}\left( {x,y} \right)} = \left\{ {\begin{matrix}{{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}} & {{\ldots \mspace{14mu} {{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}}} \leq 1} \\1 & {\ldots \mspace{14mu} {otherwise}}\end{matrix}.} \right.} & (24)\end{matrix}$

According to Equation (24), the edge strength calculation unit 235 clipsa value of ∇²(x, y)/σ²(x, y) so that ∇²(x, y)/σ²(x, y) does notexceed 1. As another example, the edge strength calculation unit 235 maynormalize the calculated value of ∇²(x, y)/σ²(x, y) so that ∇²(x,y)/σ²(x, y) does not exceed 1.

Thereafter, the edge strength unit 235 calculates, for each pixel ofinterest of an input image, the edge strength K_(G) for a pattern toneedge by multiplying the gradient ∇² of the pixel of interest by thestairs tone strength K_(S) of the pixel of interest. Here, letting thecoordinates of a pixel of interest be (x, y), the edge strength K_(G)(x, y) of the pixel of interest is calculated from the gradient ∇²(x, y)and the stairs tone strength K_(S) of the pixel of interest usingEquation (25), below:

K _(G)(x,y)=α·∇²(x,y)·K_(S)   (25)

Here, α is a constant for enhancing the edge strength K_(G). Since theedge strength K_(G) is calculated from the square of the gradient ∇,i.e., the gradient ∇², the calculated value of the edge strength K_(G)may be relatively small. Thus, the strength is enhanced by multiplyingthe gradient by α in Equation (25). As another example, as describedabove with respect to Equation (16), the edge strength calculation unit235 may calculate the edge strength K_(G) from the absolute value of thegradient ∇ and the stairs tone strength K_(S).

Therefore, the edge strength calculation unit 235 calculates, for eachpixel of interest of an input image, the edge strength K_(G) from thegradient ∇² of the pixel of interest and the stairs tone strength K_(S)of the pixel of interest. The edge strength calculation unit 235outputs, to the mixing ratio calculation unit 237, the edge strengthK_(G) for each pixel of interest of an input image.

The mixing ratio calculation unit 237 receives, from the edge strengthcalculation unit 235, the edge strength K_(G) for each pixel of interestof an input image. The mixing ratio calculation unit 237 converts theedge strength K_(G) for each pixel of interest into a mixing ratio K_(E)on the basis of a preset conversion function f_(E)(K_(G)), i.e.K_(E)=f_(E)(K_(G)). For example, FIG. 17 illustrates an example of aconversion function f_(E)(K_(G)) for performing edge strengthK_(G)-mixing ratio K_(E) conversion by the mixing ratio calculation unit237. The conversion function f_(E)(K_(G)) of FIG. 17 is expressed byEquation (26) below, where min is shown in FIG. 17:

$\begin{matrix}{{f_{E}\left( K_{G} \right)} = \left\{ \begin{matrix}1 & {{\ldots \mspace{14mu} K_{G}} < {th}_{1}} \\\min & {{\ldots \mspace{14mu} K_{G}} \geq {th}_{2}} \\{\frac{\left( {1 - \min} \right)\left( {{th}_{2} - K_{G}} \right)}{{th}_{2} - {th}_{1}} + \min} & {\ldots \mspace{14mu} {otherwise}}\end{matrix} \right.} & (26)\end{matrix}$

According to Equation (26), the mixing ratio calculation unit 237linearly modulates in an area between a maximum value of 1 and a minimumvalue min of the mixing ratio K_(E) that are functions of thresholdvalues th₁ and th₂. If α=8 for Equation (25), good illumination lightextraction may be achieved by adjusting the minimum value min and thethreshold values th₁ and th₂ such that min=0.0-0.2 and th₁,th_(e)=0.02-0.8 (th₁≦th₂).

The conversion function f_(E)(K_(G)) of FIG. 17 is merely an example,and embodiments are not limited thereto. For example, FIG. 18illustrates another example of a conversion function f_(E)(K_(G)) forperforming edge strength K_(G)-mixing ratio K_(E) conversion by themixing ratio calculation unit 237. As illustrated in FIG. 18, the mixingratio calculation unit 237 may perform edge strength K_(G)-mixing ratioK_(E) conversion using a nonlinear function.

Therefore, the mixing ratio calculation unit 237 calculates the mixingratio K_(E) for each pixel of interest from the edge strength K_(G)calculated for each pixel of interest of an input image. Furthermore,the mixing ratio calculation unit 237 outputs, to the mixing unit 21,the mixing ratio K_(E) for each pixel of interest of an input image.

The mixing unit 21 receives, from the variance calculation unit 233, theaverage value AA, a moving average, for each pixel of interest of aninput image, and, from the mixing ratio calculation unit 237, the mixingratio K_(E) for each pixel of interest of an input image. The mixingratio 21 mixes the brightness intensity of a pixel of interest with theaverage value AA of the pixel of interest based on the mixing ratioK_(E) of the pixel of interest to calculate the illumination lightcomponent LL for the pixel of interest, where each pixel of the inputimage II is sequentially processed as the pixel of interest.

Here, letting the coordinates of a pixel of interest be (x, y), and theaverage value of the pixel of interest be A(x, y), the illuminationlight component L(x, y) of the pixel of interest is calculated from themixing ratio K_(E) from Equation (27) below:

L(x,y)=(1−K _(E))·I(x,y)+K _(E) ·A(x,y)   (27)

FIG. 19 illustrates an exemplary configuration of the mixing unit 21 forcalculating the illumination light component L(x, y) from Equation (27).

As illustrated in FIG. 19, the mixing unit 21 includes, for example,multiplication units 211 and 215, a subtraction unit 213, and anaddition unit 217.

The subtraction unit 213 receives, from the mixing ratio calculationunit 237, the mixing ratio K_(E) corresponding to the pixel of interest(x, y), and outputs a difference of 1 and the mixing ratio K_(E), i.e.,1−K_(E), to the multiplication unit 211.

The multiplication unit 211 multiplies the brightness component I(x, y)of the pixel of interest of an input image by the difference 1−K_(E)received from the subtraction unit 213, and outputs the product to theaddition unit 217. The multiplication unit 211 is an example of a firstmultiplication unit, and the difference 1−K_(E) multiplied by thebrightness component I(x, y) is an example of a first coefficient.

The multiplication unit 215 multiplies the average value A(x, y) of thepixel of interest (x, y) received from the variance calculation unit 233by the mixing ratio K_(E) of the pixel of interest (x, y) received fromthe mixing ratio calculation unit 237, and outputs the product to theaddition unit 217. The multiplication unit 215 is an example of a secondmultiplication unit, and the mixing ratio K_(E) multiplied by theaverage value A(x, y) is an example of a second coefficient.

The addition unit 217 adds the product received from the multiplicationunit 211 (i.e., (1−K_(E))×I(x, y)) to the product received from themultiplication unit 215 (i.e., K_(E) X A(x, y)), and outputs the sumexpressed in Equation (27) as the illumination light component L(x, y)corresponding to the pixel of interest (x, y).

3.3. Processing

An exemplary method of operating the illumination light generation unit20 according to a present embodiment will be described with reference toFIG. 20. FIG. 20 is a flowchart that illustrates a method of operatingthe illumination light generation unit 20 according to a presentembodiment.

Operation S201

The gradient calculation unit 231 calculates the gradient ∇² for eachpixel of interest from the brightness intensities of pixels adjacent tothe pixel of interest using the gradient operator W, where each pixel ofthe input image is II sequentially processes as the pixel of interest.For example, the gradient calculation unit 231 can calculate thegradient ∇² for each pixel of interest by calculating a convolutionintegral using the gradient operator W as expressed by Equations (19) to(21). Furthermore, the gradient calculation unit 231 outputs, to theedge strength calculation unit 235, the gradient ∇² for each pixel ofinterest.

Operation S203

The variance calculation unit 233 calculates, for each pixel ofinterest, the average value AA of the brightness intensities of thepixel of interest and of pixels adjacent to the pixel of interest, whereeach pixel of the input image II is sequentially processed as the pixelof interest. The average value AA can be calculated from Equation (22).Furthermore, the variance calculation unit 233 outputs, to the mixingunit 21, the average value AA, i.e., a moving average, calculated foreach pixel of interest.

Operation S205

The variance calculation unit 233 calculates the variance σ² of thepixel of interest from the average value AA calculated for each pixel ofinterest. The variance σ² can be calculated from Equation (23).Furthermore, the variance calculation unit 233 outputs, to the edgestrength calculation unit 235, the variance σ² for each pixel ofinterest.

Operation S207

The edge strength calculation unit 235 receives, from the gradientcalculation unit 231, the gradient ∇² for each pixel of interest.Furthermore, the edge strength calculation unit 235 receives, from thevariance calculation unit 233, the variance σ² for each pixel ofinterest. The edge strength calculation unit 235 calculates thediscrepancy between the gradient ∇² and the variance σ² as the stairstone strength K_(S). The stairs tone strength K_(S) may be calculatedfrom Equation (24).

Operation S209

Thereafter, the edge strength unit 235 calculates, for each pixel ofinterest of an input image II, the edge strength K_(G)for a pattern toneedge by multiplying the gradient ∇² of the pixel of interest by thestairs tone strength K_(S) of the pixel of interest. The edge strengthK_(G) may be calculated from Equation (25). The edge strengthcalculation unit 235 outputs, to the mixing ratio calculation unit 237,the edge strength K_(G) for each pixel of interest.

Operation S211

The mixing ratio calculation unit 237 receives, from the edge strengthcalculation unit 235, the edge strength K_(G) of each pixel of interestof an input image. The mixing ratio calculation unit 237 converts theedge strength K_(G) acquired for each pixel of interest into the mixingratio K_(E) a preset conversion function f_(E)(K_(G)), such as thatexpressed by Equation (26). Furthermore, the mixing ratio calculationunit 237 outputs, to the mixing unit 21, the mixing ratio K_(E)calculated for each pixel of interest of an input image.

Operation S213

The mixing unit 21 receives, from the variance calculation unit 233, theaverage value AA, i.e., a moving average, of each pixel of interest ofan input image. Furthermore, the mixing unit 21 receives, from themixing ratio calculation unit 237, the mixing ratio K_(E) of each pixelof interest of an input image. The mixing ratio 21 combines thebrightness intensity of a pixel of interest with the average value AA ofthe pixel of interest based on the mixing ratio K_(E) of the pixel ofinterest to calculate the illumination light component LL of the pixelof interest, where each pixel of the input image II is sequentiallyprocessed as the pixel of interest. The illumination light component LLcan be calculated from Equation (27).

The above-mentioned series of operations can be performed by a programthat processes each element of the display device 10 and executes on aCPU. This program may be configured to be executed by an OS installed inthe device. A storage location of the program is not limited as long asit is readable by a device that includes a CPU for performing theabove-described processing. For example, the program may be stored in anexternal recording medium accessible from the device. In this case, therecording medium in which the program is stored may be accessed by thedevice so that the CPU of the device can execute the program.

3.4. Example

An example of an estimated circuit size of the illumination lightgeneration unit 20 according to a present embodiment will be described.An example of the estimated circuit size of the illumination lightgeneration unit 10′, i.e., comparative example 1, described above withreference to FIG. 3 and an example of the estimated circuit size of theillumination light generation unit 10, i.e., comparative example 2,according to an embodiment described above with reference to FIG. 7 willalso be described. In a present description, the circuit size may beestimated based on the number of gates in a gate circuit. For example,estimated circuit size reference values are “104” for an adder and asubtracter, “384” for a multiplier, “1600” for a divider, “64” for anabsolute value calculation circuit, and “40” for a 2-input selector.

FIG. 21 is a diagram that illustrates an estimated circuit size of anexemplary illumination light generation unit according to a presentembodiment and the estimated circuit sizes of illumination lightgeneration units according to comparative examples 1 and 2.

As illustrated in FIG. 21, an illumination light generation unit ofcomparative example 1 includes an ε filter, an ε value adjusting unit,and a gradient calculation unit, and the ε filter has a largest circuitsize, about 3800. Compared to an illumination light generation unit ofcomparative example 1, an illumination light generation unit ofcomparative example 2 further includes a variance calculation unit andan edge strength calculation unit, and has a circuit size larger thanthat of an illumination light generation unit of comparative example 1due to the circuit sizes of the additional elements.

The ε filter and the ε value adjusting unit of an illumination lightgeneration unit of comparative example 2 are replaced with a mixing unitand a mixing ratio calculation unit in an illumination light generationunit of an example illustrated in FIG. 16. The ε value adjusting unit isapproximately equivalent to the mixing ratio calculation unit in termsof a circuit size, but the circuit size of the mixing unit, see FIG. 19for example, is about 800, less than that of the ε filter.

Therefore, an illumination light generation unit according to an examplecan enable a reduced circuit size compared to a illumination lightgeneration unit of comparative example 2. In detail, on the basis of thecircuit size of an illumination light generation unit of comparativeexample 1, the circuit size of the illumination light generation unit ofcomparative example 2 increases by about 60%, but the circuit size of anillumination light generation unit of an example increases by about 20%.As described above, like an illumination light generation unit 10 of acomparative example 2 according to an embodiment of FIG. 7, anillumination light generation unit 20 of an example according to anembodiment of FIG. 16 can differentiate a stairs tone edge from apattern tone edge in an input image, thereby enabling nonlinearsmoothing processing for the input image.

3.5. Summary

As described above, the illumination light generation unit 20 accordingto an embodiment of the inventive concept calculates the gradient ∇² andthe variance σ² for each pixel of interest, and calculates the stairstone strength K_(S) from the discrepancy between the gradient ∇² and thevariance σ². The illumination light generation unit 20 calculates theedge strength K_(G) from the stairs tone strength K_(S), and calculatesthe mixing ratio K_(E) from the edge strength K_(G). Furthermore, theillumination light generation unit 20 mixes the brightness intensity ofa pixel of interest with the average value AA, which is a movingaverage, of the brightness intensities of the pixel of interest and thepixels adjacent to the pixel of interest based on the mixing ratioK_(E), to generate the illumination light component LL.

By virtue of a configuration according to an embodiment of the inventiveconcept, like the illumination light generation unit 10 according to anembodiment of FIG. 7, the illumination light generation unit 20according to a present embodiment can differentiate a stairs tone edgefrom a pattern tone edge in an input image, thereby enabling nonlinearsmoothing processing for the input image.

Furthermore, in the illumination light generation unit 20 according to apresent embodiment, a relatively large circuit, such as an ε filter, canbe replaced with a smaller circuit, such as the mixing unit 21.Accordingly, the illumination light generation unit 20 according to apresent embodiment can reduce a circuit size compared to theillumination light generation unit 10 according to an embodiment.

In particular, a change in the illumination light component is small andsubstantially linear within a reference pixel range of the operatorlength n=5i of a high resolution device, such as a high definition (HD),full high definition (FHD), ultra high definition (UHD), etc. Therefore,on the basis of such characteristics, the illumination light generationunit 20 according to a present embodiment calculates the illuminationlight component LL using a moving average value AA of the brightnessintensities II of pixels within a predetermined reference pixel rangebased on a pixel of interest, so that the circuit size may be reduced.

Moreover, the illumination light generation unit 20 according to apresent embodiment reuses the average value AA calculated duringderivation of the variance G², to calculate the illumination lightcomponent LL from the mixing ratio K_(E). Therefore, since a newlycalculated average value is not used to calculate the illumination lightcomponent LL, the circuit size may be further reduced.

As described above, embodiments of the inventive concept provide animage processing device, an image processing method, and a program fordifferentiating a stairs tone edge from a pattern tone edge in an imageto perform a nonlinear smoothing operation on the image.

The above-disclosed subject matter is to be considered illustrative andnot restrictive, and the appended claims are intended to cover all suchmodifications, enhancements, and other embodiments, which fall withinthe true spirit and scope of embodiments of the inventive concept. Thus,to the maximum extent allowed by law, the scope of embodiments of theinventive concept is to be determined by the broadest permissibleinterpretation of the following claims and their equivalents, and shallnot be restricted or limited by the foregoing detailed description.

What is claimed is:
 1. An image processing device comprising: an edgestrength calculation unit that calculates an edge strength of a pixel ofinterest in an input image from a stairs tone strength of the pixel ofinterest and a gradient of the pixel of interest, wherein the gradientis calculated from pixel values of the pixel of interest and pixelsadjacent to the pixel of interest, and the stairs tone strength isindicative of differences between the gradient and a variance of thepixel of interest calculated from the pixel values of the pixel ofinterest and the adjacent pixels; and a mixing unit that calculates anillumination light component of the pixel of interest of the input imagefrom the pixel value of the pixel of interest, an average value of thepixel values of the pixels adjacent to the pixel of interest, and amixing ratio of the pixel of interest calculated from the edge strength.2. The image processing device of claim 1, further comprising: avariance calculation unit that calculates the average value from thepixel values of the of the pixel of interest and adjacent pixels withina predetermined range of the pixel of interest, and calculates thevariance from the average value.
 3. The image processing device of claim1, further comprising: a gradient calculation unit that calculates thegradient of the pixel of interest from pixel values of the pixel ofinterest and adjacent pixels within a predetermined range of the pixelof interest.
 4. The image processing device of claim 1, furthercomprising: a mixing ratio calculation unit that calculates the mixingratio of the pixel of interest from the edge strength of the pixel ofinterest.
 5. The image processing device of claim 4, wherein the mixingratio K_(E) is calculated from${f_{E}\left( K_{G} \right)} = \left\{ {\begin{matrix}1 & {{\ldots \mspace{14mu} K_{G}} < {th}_{1}} \\\min & {{\ldots \mspace{14mu} K_{G}} \geq {th}_{2}} \\{\frac{\left( {1 - \min} \right)\left( {{th}_{2} - K_{G}} \right)}{{th}_{2} - {th}_{1}} + \min} & {\ldots \mspace{14mu} {otherwise}}\end{matrix},} \right.$ wherein K_(G) is the edge strength, f_(E)(K_(G))is a function that converts the edge strength K_(G) into the mixingratio K_(E), th₁ and th₂ are threshold values of the edge strength,wherein th₁<th₂, and min is a preset constant greater than zero.
 6. Theimage processing device of claim 1, wherein the stairs tone strengthK_(S)(x, y) of the pixel of interest is calculated from${K_{s}\left( {x,y} \right)} = \left\{ {\begin{matrix}{{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}} & {{\ldots \mspace{14mu} {{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}}} \leq 1} \\1 & {\ldots \mspace{14mu} {otherwise}}\end{matrix},} \right.$ wherein ∇²(x, y) is the gradient of the pixel ofinterest, and σ²(x, y) is the variance of the pixel of interest, and theedge strength K_(G)(x, y) of the pixel of interest is calculated fromK _(G)(x,y)=α·∇²(x,y)·K _(S), wherein α is a predetermined constant. 7.The image processing device of claim 1, wherein the mixing unitcomprises: a first multiplication unit that multiplies the pixel valueof the pixel of interest by a first coefficient based on the mixingratio; a second multiplication unit that multiplies the average value bya second coefficient based on the mixing ratio; a subtraction unit thatthat calculates that first coefficient by subtracting the mixing ratiofrom one; and an addition unit that adds an output of the firstmultiplication unit to an output of the second multiplication unit. 8.The image processing device of claim 1, further comprising: a divisionunit that receives the illumination light component from theillumination light generation unit and that calculates a reflectivitycomponent of the input image by dividing the input image by theillumination light component; an illumination light modulation unit thatlocally modulates the illumination light component to generate a newillumination light component; and a multiplication unit that multipliesthe reflectivity component received from the division unit by the newillumination light component received from the illumination lightmodulation unit to generate a brightness component.
 9. A method ofprocessing an image, comprising the steps of: calculating an edgestrength of a pixel of interest in an input image from a stairs tonestrength of the pixel of interest and a gradient of the pixel ofinterest, wherein the gradient is calculated from pixel values of thepixel of interest and pixels adjacent to the pixel of interest, and thestairs tone strength is indicative of differences between the gradientand a variance of the pixel of interest calculated from the pixel valuesof the pixel of interest and the adjacent pixels; and calculating anillumination light component of the pixel of interest of the input imagefrom the pixel value of the pixel of interest, an average value of thepixel values of the pixels adjacent to the pixel of interest, and amixing ratio of the pixel of interest calculated from the edge strength.10. The method of claim 9, wherein the average value is calculated fromthe pixel values of the pixel of interest and adjacent pixels within apredetermined range of the pixel of interest, and the variance iscalculated from the average value.
 11. The method of claim 9, whereinthe gradient is calculated from pixel values of the pixel of interestand adjacent pixels within a predetermined range of the pixel ofinterest.
 12. The method of claim 9, wherein the mixing ratio K_(E) ofthe pixel of interest is calculated from the edge strength of the pixelof interest using${f_{E}\left( K_{G} \right)} = \left\{ {\begin{matrix}1 & {{\ldots \mspace{14mu} K_{G}} < {th}_{1}} \\\min & {{\ldots \mspace{14mu} K_{G}} \geq {th}_{2}} \\{\frac{\left( {1 - \min} \right)\left( {{th}_{2} - K_{G}} \right)}{{th}_{2} - {th}_{1}} + \min} & {\ldots \mspace{14mu} {otherwise}}\end{matrix},} \right.$ wherein K_(G) is the edge strength, f_(E)(K_(G))is a function that converts the edge strength K_(G) into the mixingratio K_(E), th₁ and th₂ are threshold values of the edge strength,wherein th₁<th₂, and min is a preset constant greater than zero.
 13. Themethod of claim 9, wherein the stairs tone strength K_(S)(x, y) of thepixel of interest is calculated from${K_{s}\left( {x,y} \right)} = \left\{ {\begin{matrix}{{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}} & {{\ldots \mspace{14mu} {{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}}} \leq 1} \\1 & {\ldots \mspace{14mu} {otherwise}}\end{matrix},} \right.$ wherein ∇²(x, y) is the gradient of the pixel ofinterest, and σ²(x, y) is the variance of the pixel of interest, and theedge strength K_(G)(x, y) of the pixel of interest is calculated fromK _(G)(x,y)=α·∇²(x,y)·K _(S), wherein α is a predetermined constant. 14.The method of claim 9, wherein the illumination light component L(x, y)of the pixel of interest is calculated fromL(x,y)=(1−K _(E))·I(x,y)+K _(E) ·A(x,y), wherein K_(E) is the mixingratio, I(x, y) is the pixel value of the input image at the pixel ofinterest, and A(x, y) is the average value of the pixel of interest. 15.A non-transitory program storage device readable by a computer, tangiblyembodying a program of instructions executed by the computer to performthe method steps for processing an image, the method comprising thesteps of calculating an edge strength of a pixel of interest in an inputimage from a stairs tone strength of the pixel of interest and agradient of the pixel of interest, wherein the gradient is calculatedfrom pixel values of the pixel of interest and pixels adjacent to thepixel of interest, and the stairs tone strength is indicative ofdifferences between the gradient and a variance of the pixel of interestcalculated from the pixel values of the pixel of interest and theadjacent pixels; and calculating an illumination light component of thepixel of interest of the input image from the pixel value of the pixelof interest, an average value of the pixel values of the pixels adjacentto the pixel of interest, and a mixing ratio of the pixel of interestcalculated from the edge strength.
 16. The computer-readable storagedevice of claim 15, wherein the average value is calculated from thepixel values of the pixel of interest and adjacent pixels within apredetermined range of the pixel of interest, and the variance iscalculated from the average value.
 17. The computer-readable storagedevice of claim 15, wherein the gradient is calculated from pixel valuesof the pixel of interest and adjacent pixels within a predeterminedrange of the pixel of interest.
 18. The computer-readable storage deviceof claim 15, wherein the mixing ratio K_(E) of the pixel of interest iscalculated from the edge strength of the pixel of interest using${f_{E}\left( K_{G} \right)} = \left\{ {\begin{matrix}1 & {{\ldots \mspace{14mu} K_{G}} < {th}_{1}} \\\min & {{\ldots \mspace{14mu} K_{G}} \geq {th}_{2}} \\{\frac{\left( {1 - \min} \right)\left( {{th}_{2} - K_{G}} \right)}{{th}_{2} - {th}_{1}} + \min} & {\ldots \mspace{14mu} {otherwise}}\end{matrix},} \right.$ wherein K_(G) is the edge strength, f_(E)(K_(G))is a function that converts the edge strength K_(G) into the mixingratio K_(E), th₁and th₂ are threshold values of the edge strength,wherein th₁<th₂, and min is a preset constant greater than zero.
 19. Thecomputer-readable storage device of claim 15, wherein the stairs tonestrength K_(S) (x, y) of the pixel of interest is calculated from${K_{s}\left( {x,y} \right)} = \left\{ {\begin{matrix}{{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}} & {{\ldots \mspace{14mu} {{\nabla^{2}\left( {x,y} \right)}/{\sigma^{2}\left( {x,y} \right)}}} \leq 1} \\1 & {\ldots \mspace{14mu} {otherwise}}\end{matrix},} \right.$ wherein ∇²(x, y) is the gradient of the pixel ofinterest, and σ²(x, y) is the variance of the pixel of interest, and theedge strength K_(G)(x, y) of the pixel of interest is calculated fromK _(G)(x,y)=α·∇²(x,y)·K _(S), wherein α is a predetermined constant. 20.The computer-readable storage device of claim 15, wherein theillumination light component L(x, y) of the pixel of interest iscalculated fromL(x,y)=(1−K _(E))·I(x,y)+K _(E) ·A(x,y), wherein K_(E) is the mixingratio, I(x, y) is the pixel value of the input image at the pixel ofinterest, and A(x, y) is the average value of the pixel of interest.